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DETAILED ACTION 

Response to Amendment 

1. Applicant's amendment filed on October 13, 2009 has been considered and entered in full. 

2. Information disclosure statement (IDS) filed on 10/13/2009 has been considered and 
entered in full. 

3. Applicant's amendment to the specification has been considered and entered in full. Based 
on the applicant's amendment to the specification, claim rejections under 35 USC 101 have been 
withdrawn. 

4. Applicant's arguments with respect to amended claims have been considered but are moot in 
view of the ground(s) of rejection(s) made below. 

Response to Arguments 

5. Applicant's arguments regarding the prior art under Hatfield and Newman on pages 11-12 of 
the amendment filed on October 13, 2009 have been fully considered but are not persuasive. 
Applicant argues in substance that "Neither Hatfield nor Newman discloses dynamically modifying a 
characteristic of a graphical user interface of the application based, art least in part, on the 
interpretation of the eye tracking data and the external context data". Examiner respectfully 
disagrees. As cited in the previous office action, Hatfield and Newman both disclose windows 
operating system environment and windows operating environment is nothing but a graphical user 
interface based environment. Now regarding the dynamic environment, Hatfield, as discussed 
before in the rejections, discloses synchronizing eye and voice with the cursor and then controlling 
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the system and this can only be done using dynamic applications and the same is taught by Newman. 
Now regarding "dynamically modifying a characteristic of a graphical user interface of the 
application, based, at least in part, on the interpretation of the eyetracking data and the external 
context data", Hatfield discloses (Topic 4 - Initial EVA Implementation - page 4. last few lines 
through page 5 - in internet explorer, fixate a particular hyperlink and utter "click" which has the 
effect of clicking with a mouse on the hyperlink, thus hyperlink being the characteristic of the 
application (internet explorer) and dynamically acting on the characteristic being clicking on the 
hyperlink to invoke the internet explorer to display the contents of the link on the browser and after 
clicking, it opens a new link in the browser which being dynamically modifying a characteristic of a 
graphical user interface of the application). Similarly, Newman discloses (a host computer could also 
be responsive to several users having headsets that may be working as a team to repair the piece of 
equipment. The users could all work form the same procedure or call for a different procedure to 
be displayed as an additional window in the display" (col. 7, lines 1-15)), where displaying a 
different procedure to be displayed being dynamically modifying a characteristic of a graphical user 
interface of the application. Thus, the combined invention of Hatfield and Newman still discloses 
the claimed invention as presented in the amended claims. 



Claim Rejections - 35 USC § 112 

6. The following is a quotation of the second paragraph of 35 U.S.C. 112: 

The specification shall conclude with one of more claims particularly pointing out and distinctly claiming the subject 
m:mcr which the applicant regards as his invention. 

7. Claims 2, 8, 10, 13 and 16 rejected under 35 U.S.C. 112, second paragraph, as being 
indefinite for failing to particularly point out and distinctly claim the subject matter which applicant 
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regards as the invention. Claim 2 depends on claim 1 and claim 2 recites the limitation "wherein 
acting on comprises one of:". There is insufficient antecedent basis for this limitation in the claim. 

Claim 8 depends on claim 7 and claim 8 recites the limitation "wherein acting on comprises 
one of:". There is insufficient antecedent basis for this limitation in the claim. 

Claim 10 depends on claim 9 and claim 10 recites the limitation "wherein acting on 
comprises one of:". There is insufficient antecedent basis for this limitation in the claim. 

Claim 13 depends on claim 12 and claim 13 recites the limitation "wherein acting on 
comprises one of:". There is insufficient antecedent basis for this limitation in the claim. 

Claim 16 depends on claim 15 and claim 16 recites the limitation "wherein acting on 
comprises one of:". There is insufficient antecedent basis for this limitation in the claim. 

Claim Rejections - 35 USC § 103 

8. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness 
rejections set forth in this Office action: 

(a) A patent tray not he obtained though the invention is not identically disclosed or described as set forth in section 
1 02 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the 
subject matin- as a whole would have been obvious at the time the invention was made to a person having ordinary 
skill in the art to which said subject matter pertains. Patcntabiliu shall not be negatived by the manner in which the 

9. Claims 1-2, 4-5, 7-10, 12-13 and 15-22 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Hatfield et al., November 30, 1997, "An Interface integrating eye gaze and voice 
recognition for hands-free computer access" (pages 1-7), further in of view of Newman et al., U.S. 
Patent No. 5,844,824. 
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Hatfield discloses an EVA (Eye/Voice Aware) interface which allows combined eye-tracking 
and voice input to allow hands-free interaction with a standard window-based computer application, 
where voice input being the external context data. 

Hatfield discloses "In EVA, we implemented a mode of operation where the user can slave the 
screen cursor to his/her eye. Wherever the user looks on the display while operating in this mode, 

the cursor will be displayed. . .The user fixates a Windows button The system will detect rapid 

eye movement, e.g., saccades and adjust very quickly to gross changes in POG "(Hatfield, topic 4 - 
Initial EVA Implementation, pages 4 and 5). Clearly from Hatfield's disclosure, it is clear that the 
cursor is a slave of the eye, and the cursor motion is defined by the eye movement (or eye tracking 
data) and it being apparent that in order to interpret the eye motion (or eye tracking data), an eye 
interpretation engine (or system or component) is inherendy required which would send the 
translated data to the system to synchronize the cursor with the eye motion (or eye behavior), 
without which EVA system won't be able to synchronize the eye with cursor. The interpretation 
component is a must part of a system which uses eye motion or eye behavior to control a system. 
Thus, EVA system, receiving from an eye interpretation engine, at least an interpretation of 
eye tracking data corresponding to at least one user. 

Hatfield further discloses receiving external context data corresponding the at least one 
user's interaction with an application (Topic 4 - Initial EVA Implementation- pages 4 and 5 - 
external context data being voice commands such as "forward tab", "click", "page up" and "page 
down" — With IE users can navigate the various hyperlinks in a web page by "tabbing between 
them" with the command "forward tab". They can also fixate a particular hyperlink and utter 
"click", page 4 -last few lines). Now, as in Microsoft Windows, the operations of the computing 
device running the application are controlled by the use of cursor and the buttons, and controlling 
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the system is basically controlling the operational state of the processor of the computing device. 
Hatfield discloses that voice input is used as command entry for controlling the operational states of 
the processor of the computer running the application. For example, voice commands such as 
"forward tab, "page up" and "page down" would control the operational state of browsing of 
internet explorer in the device (pages 4, last few lines through page 5), which conforms to the 
limitation "wherein the external context data is related to an operational state associated with a 
computing device running the application". 

Hatfield further discloses "based, at least in part, on the interpretation of the eye tracking 
data and the external context data, dynamically modifying a characteristic of a graphical user 
interface of the application" (Topic 4 - Initial EVA Implementation - page 4. last few lines through 
page 5 - in internet explorer, fixate a particular hyperlink and utter "click" which has the effect of 
clicking with a mouse on the hyperlink, thus hyperlink being the characteristic of the application 
(internet explorer) and dynamically acting on the characteristic being clicking on the hyperlink to 
invoke the internet explorer to display the contents of the link on the browser and after clicking, it 
opens a new link in the browser which being dynamically modifying a characteristic of a graphical 
user interface of the application). 

As discussed before, Hatfield discloses an EVA (Eve/Voice Aware) interface which allows 
combined eye-tracking and voice input to allow hands-free interaction with a standard window- 
based computer application, where voice input being the external context data. Hatfield provides an 
example of an application in which the combination of eyetracking and voice commands is used by a 
single user of the computer but does not expressly teach the usage of the combination of 
eyetracking and voice by a plurality of users. 
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However, Newman discloses a handfree system in which the user may operate the 
computing apparatus to display information in a hands-free manner utilizing eye tracking with or 
without audio commands (col. 2, lines 61-64). Newman further discloses "the system concept of 
eye-tracking permits the operator to manipulate the computer completely hands-free by tracking the 
eye and interpreting its movements as "mouse" commands to the application. Generally, the eye 
tracking system is head-mounted on the user but may be used in any suitable manner" (col. 3, lines 
40-45). Newman further discloses "a voice command can be given to the present computer which is 
confirmed or enhanced by the present eye-tracking system and vice-versa" (col. 3, lines 42-45). 
Newman further discloses "the system can work in two modes: In the first mode, a user having 
a headset with activation means, display screen and communication means is linked with a host 
computer either worn on the user's person contained within the display means or remote from the 
person. The user would command the host computer to display a particular procedure for repairing 
a piece of equipment. The host computer would then search for an transmit the procedure back to 
the user for display on the user's headset. In the second mode, a host computer could also be 
responsive to several users having headsets that may be working as a team to repair the 
piece of equipment. The users could all work from the same procedure or call for a different 
procedure to be displayed as an additional window in the display (col. 7, lines 1-15). 
Therefore, it would have been obvious to one of ordinary skill in the art at the time invention was 
made to combine the teachings of Newman of extending the combination of eye tracking and voice 
data to multiple-users to control/ operate the computer with that of Hatfield. One of ordinary skill in 
the art would have been motivated to combine the teachings of Newman with that of Hatfield 
because both references are directed to the same field of endeavor and Newman's teachings when 
combined with the teachings of Hatfield would enhance or provide the Hatfield's invention with 
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additional capability or versatility of handling applications where multiple users concurrently use the 
computer to share the same application when working on the same task, and would further increase 
the efficiency of the group or users working on the task. 

Regarding claim 2, claim 2 recites wherein acting on comprises one of: modifying, creating, 
destroying, removing, invoking and configuring. Hatfield discloses "in internet explorer, fixate a 
particular hyperlink and utter "click" which has the effect of clicking with a mouse on the 
hyperlink", (Topic 4 - Initial EVA Implementation - page 4. last few lines through page 5), thus 
hyperlink being the characteristic of the application (internet explorer) and dynamically acting on the 
characteristic being clicking on the hyperlink to invoke the internet explorer to display the contents 
of the link on the browser. 

Regarding claim 4, all the limitations in the claim 4 are similar to claim 1 except that claim 4 
recites "dynamically acting on a graphical output of the application displayed on an output device 
based, at least in part, on the interpretation of the eye tracking data and the external context data". 
As discussed before, Hatfield discloses (Topic 4 - Initial EVA Implementation - page 4. last few 
lines through page 5 - in internet explorer, fixate a particular hyperlink and utter "click" which has 
the effect of clicking with a mouse on the hyperlink) thus hyperlink being the characteristic of the 
application (internet explorer) and dynamically acting on the characteristic being clicking on the 
hyperlink to invoke the internet explorer to display the contents of the link on the browser. Thus 
clicking on the hyperlink would invoke the internet explorer to display the contents of the link as 
output on the browser, where the output again being a page in the internet explorer with hyperlink. 
Hatfield further discloses acting on the output (page 5, topic 4, 1 st few lines - Users can also scroll 
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the browser by issuing "page up" and "page down" commands, open list boxes and select various 
items). 

Claim 5 has been similarly analyzed and rejected as per claim 2. 

Claims 7-10, 12-13 and 15-16 have been similarly analyzed and rejected as per claims 1-2 and 

4-5. 

Regarding claims 17-22, Hatfield discloses the external sources (external context data) 
comprise one or more of: system information, location and/or z-order of windows and/or objects, 
document Object Model (DOM) of a Web page or application being viewed, current application 
process state and/or visual state, task models, cognitive models describing the mental or physical 
steps or states required (page 4 - last few lines - current application process state and/ or visual 
state - tabbing between the hyperlinks which being the process/ visual state of browsing, the tasks 
model including tabbing and scrolling, fixating and clicking a windows button). 

10. Claims 3, 6, 11 and 14 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Hatfield et al., November 30, 1997, "An Interface integrating eye gaze and voice recognition for 
hands-free computer access" (pages 1-7), further in of view of Newman et al., U.S. Patent No. 
5,844,824, and further in view of Martins, U.S. Patent No. 6,608,615 Bl. 

Regarding claims 3, 6, 11 and 14, as discussed in the rejection of claims 1 and 4, the 
combined invention of Hatfield and Newman discloses that eye tracking data can be used to control 
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various aspects of the interface for example controlling the browsing in internet explorer (IE) and 
selecting the various items in IE but combined invention of Hatfield and Newman does not 
explicitly disclose the extended applications where the eye tracking data would be used such as 
"determining from the interpretation of the eye tracking data at least a portion of an interface that 
has not been viewed by the user and modifying a format of the portion of the interface that has not 
been viewed by the user". However, Martins which belong to the same field of endeavor describes 
the extended applications where the eye tracking data can be used. Martins discloses "if the banner 
add is not viewed at all, the intervention would cause the banner to dvnamicallv appear to jump 
closer to the user's gaze" (Martins, col. 6, lines 19-21). Therefore, it would have been obvious for 
one of ordinary skill in the art at the time of invention was made to combine the teachings of 
Martins with that of combined invention of Hatfield and Newman. One of ordinary skill in the art 
would have been motivated to combine the teachings of Martins with that of combined invention of 
Hatfield and Newman as it would enhance the capabilities of combined invention of Hatfield and 
Newman to be used in extended applications, thus making Hatfield's system more versatile in the 
field of controlling the interface application using eye tracking data. 

Examiner note: Examiner has cited particular columns and line numbers in the references as 
applied to the claims above for the convenience of the applicant. Although the specified citations are 
representative of the teachings for the art and are applied to the specific limitations within the 
individual claim, other passages and figures may be applied as well. It is respectfully requested from 
the applicant in preparing responses, to fully consider the references entirely as potential teachings 
all or part of the claimed invention, as well as the context of the passage as taught by the prior art or 
disclosed by the examiner. 
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Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is 
reminded of the extension of time policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 

Any inquiry concerning this communication or earlier communications from the examiner 
should be directed to Manav Seth whose telephone number is (571) 272-7456. The examiner can 
normally be reached on Monday to Friday from 8:30 am to 5:00 pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, 
Vikkram Bali, can be reached on (571) 272-7415. The fax phone number for the organization where 
this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR system, 
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see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, 
contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 



/Manav Seth/ 

Primary Examiner, Art Unit 2624 
November 8, 2009 



